Feature/env variables #127
Conversation
added functionality to identify environment variables enclosed in ${..} with a regex
This feature enables the yarrrml-parser to read content of a `.env` file and replaces dollar variables with curly braces in a YARRRML document accordingly. Thus the yarrrml-parser follows the common practice to define secrets in a `.env` file. Mapping files will be more reusable as they can be used in environments with different secrets or any other replacable value such as sources, targets etc. Three test cases are provided to test that all variables are replaced and that warnings are emitted when not all variables can be replaced because no environment variable was specified.
| @@ -37,6 +37,12 @@ You find an example in [`test/template-escape`](test/template-escape). | |||
| If you want to use for example `$(_name)` as both an external reference and a normal reference, | |||
| then you add a `\` for the latter resulting in `$(\_name)` for the latter. | |||
|
|
|||
| If your YARRRML document contains sensitive information such as database credentials, | |||
| you can use dollar variables with curly braces and specify a variable in a `.env` file. | |||
| For example, use `${DB_PASSWORD}` in the YARRRML document and add a `.env` file with content `DB_PASSWORD=mySecretPassword`. | |||
There was a problem hiding this comment.
Why can't we use normal brackets as well here? And check all variables if they are defined in the .env?
There was a problem hiding this comment.
So you mean $(DB_PASSWORD)?
This would clash with YARRRML templates: https://rml.io/yarrrml/spec/#template
Thus if you would (for some reason) specify value=myValue in your .env then also yarrrml templates $(value) would be replaced.
Similarly, if you would define VALUE=myValue in .env then a potential yarrrml template $(VALUE) would be replaced.
Such a clash with YARRRML templates is in my opinion not intended, because they reflect e.g. column names in a source.
Furthermore, curly braces (or no braces) are the common way to represent (environment) variables in UNIX, regular brackets are used for sub-shells: https://dev.to/rpalo/bash-brackets-quick-reference-4eh6
There was a problem hiding this comment.
Or do the same as with the external references? Adding a _ before the variable name? so you would have then $(_DB_PASSWORD)? So external references can come from both the CLI and .env, which makes sense.
There was a problem hiding this comment.
This would technically work but would be a convention which has to be used just because of YARRRML and therefore would harm reusability/interoperability in cases where YARRRML is not the only component.
There might be other components using .env variables, for example a script which creates an SSH tunnel to a remote DB host behind a firewall, some UI component or any other application doing something.
IMHO YARRRML should integrate into an existing setup as easy as possible. A setup in which certain environment variables are already in place and are used by other systems. As a user I would not want to change my whole application the moment I want to integrate YARRRML mappings. This could be a potential pitfall preventing people from using YARRRML.
As I see it, reducing the expressibility of environment variables (only with prefixed underscore) just to combine it with external references (which already can be provided via parameter) would limit this feature.
There was a problem hiding this comment.
You only have to change your YARRRML files, nothing else. The variables names remain the same. You only add the _ in the YARRRML template.
Added feature to use a .env file, including test cases
This feature enables the yarrrml-parser to read content of a
.envfile and replaces dollar variables with curly braces in a YARRRML document accordingly. Thus the yarrrml-parser follows the common practice to define secrets in a.envfile.Mapping files will be more reusable as they can be used in environments with different secrets or any other replaceable value such as sources, targets etc.
Three test cases are provided to test that all variables are replaced and that warnings are emitted when not all variables can be replaced because no environment variable was specified.